Apparatus and method for calibration of gaze detection

ABSTRACT

There is provided an information processing apparatus including an operation detecting unit configured to detect an operation of an operator to an object that is displayed in a display screen image for performing a predetermined input, a sight line detecting unit configured to detect a movement of a sight line of the operator on the display screen image, and a correction coefficient acquiring unit configured to acquire a correction coefficient for correcting an error of a case where the operator performs sight line input, on the basis of the movement of the sight line detected during the operation of the operator to the object.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 14/608,509 filed Jan. 29, 2015 now issued as a U.S.Pat. No. 9,684,373 on Jun. 20, 2017, which claims the benefit ofJapanese Priority Patent Application JP 2014-0233301 filed Feb. 10,2014, the entire contents of which are incorporated herein by reference.

BACKGROUND

The present disclosure relates to an information processing apparatus,an information processing method, and a program.

A technology for detecting a sight line of a person, for example,projects an infrared light or the like on an eyeball of a user, anddetects the sight line from the pupil center and the corneal curvaturecenter obtained from the position of the reflected image on the cornealsurface. This technology is utilized to determine the position that theuser gazes at on a display.

In the meantime, an error sometimes occurs between the position on thedisplay which is determined by the sight line detected by utilizing thereflected image in the corneal surface, and the position that the useractually gazes at. In order to correct this error, the sight linecalibration is executed to calculate a correction coefficient forcompensating the error (refer to JP 2012-65781A).

SUMMARY

However, the sight line calibration, for example, sequentially displaysa plurality of gazing points on the display, to prompt the user to gazeat the gazing point. Therefore, the burden on the user is likely toincrease.

Therefore, the present disclosure proposes a method to execute the sightline calibration without making the user conscious of the load of thesight line calibration.

According to an embodiment of the present disclosure, there is providedan information processing apparatus including an operation detectingunit configured to detect an operation of an operator to an object thatis displayed in a display screen image for performing a predeterminedinput, a sight line detecting unit configured to detect a movement of asight line of the operator on the display screen image, and a correctioncoefficient acquiring unit configured to acquire a correctioncoefficient for correcting an error of a case where the operatorperforms sight line input, on the basis of the movement of the sightline detected during the operation of the operator to the object.

According to another embodiment of the present disclosure, there isprovided an information processing method including detecting anoperation of an operator to an object that is displayed in a displayscreen image for performing a predetermined input, detecting a movementof a sight line of the operator on the display screen image, andacquiring a correction coefficient for correcting an error of a casewhere the operator performs sight line input, on the basis of themovement of the sight line detected during the operation of the operatorto the object.

According to another embodiment of the present disclosure, there isprovided a program for causing a computer to execute detecting anoperation of an operator to an object that is displayed in a displayscreen image for performing a predetermined input, detecting a movementof a sight line of the operator on the display screen image, andacquiring a correction coefficient for correcting an error of a casewhere the operator performs sight line input, on the basis of themovement of the sight line detected during the operation of the operatorto the object.

As described above, according to the present disclosure, the sight linecalibration is executed without making the user conscious of the load ofthe sight line calibration.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram for describing an example of sight linedetection;

FIG. 2 is a diagram illustrating a device used in sight line detectionby an infrared light;

FIG. 3 is a schematic diagram for describing calibration according to acomparative example;

FIG. 4 is a flowchart illustrating a procedure of calibration accordingto a comparative example;

FIG. 5 is a block diagram illustrating an example of a function andconfiguration of an information processing apparatus according to anembodiment of the present disclosure;

FIG. 6 is a diagram illustrating an example of a display screen imagefor UI operation, including an operation object;

FIG. 7 is a diagram for describing a cross-correlation function whenwaveforms of two time-series data functions are similar to each other;

FIG. 8 is a diagram for describing a cross-correlation function whenwaveforms of two time-series data functions are not similar to eachother;

FIG. 9 is a diagram illustrating an example of a saliency map withrespect to a display screen image including an operation object;

FIG. 10 is a flowchart illustrating an example of an operation of aninformation processing apparatus according to an embodiment;

FIG. 11 is a diagram illustrating a distance between a sight lineposition and a gravity center position of an operation object;

FIG. 12 is an explanatory diagram illustrating an example of a hardwareconfiguration of an information processing apparatus; and

FIG. 13 is a diagram illustrating an example of a function andconfiguration of an information processing apparatus provided in aserver.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Note that description will be made in the following order.

1. Sight Line Detection and Calibration

1-1. Overview of Sight Line Detection

1-2. Calibration according to Comparative Example

2. First Embodiment

2-1. Configuration of Information Processing Apparatus

2-2. Operation of Information Processing Apparatus

3. Second Embodiment

4. Hardware Configuration

5. Conclusion

1. Sight Line Detection and Calibration

(1-1. Overview of Sight Line Detection)

Various methods are proposed as the method to detect the sight line of auser.

FIG. 1 is a schematic diagram for describing an example of the sightline detection. Here, as illustrated in FIG. 1, the eye of the userlooks at a display screen of a display unit 11 of an information device10. First, the information device 10 projects an infrared light on theeye of the user. Thereafter, the information device 10 captures areflected light of the infrared light with an imaging unit 13 such as acamera. Thereby, the position that the user looks at, which is the sightline, is detected and tracked. Here, the sight line means the point onthe display unit 11 that the eye of the user looks at.

Although in FIG. 1 the infrared light is directly projected on the eyeof the user, the user may wear a device 20 of a glasses type illustratedin FIG. 2, for example. FIG. 2 is a diagram illustrating an example ofthe device 20 used in the sight line detection by the infrared light.

Although in the above an example in which the sight line is detectedusing the infrared light is described, the present disclosure is notlimited thereto. For example, there is a method to detect and track thesight line, using only the shot image captured by the imaging unit,without projecting the infrared light.

Also, as the estimation method of the sight line, there is a method toestimate the sight line using a three-dimensional model of an eyeball,for example. This estimation method of the sight line is performed as inthe following.

First, the position and the direction (the optical axis, the eye axis)of the eyeball relative to the display screen is detected, using theeyeball image or the like captured by the imaging unit. Next, thedifference (the error) in relation to the sight line (the visual axis)of the user is corrected (i.e., the correction of the error between theoptical axis and the visual axis), and estimated from the detectedposition and direction of the eyeball. Next, the point that the userlooks at in the display screen is estimated from the intersection pointof the visual axis and the display screen.

(1-2. Calibration According to Comparative Example)

In the meantime, in order to accurately track the sight line, it isimportant to adequately correct the error between the optical axis andthe visual axis. That is, what is called the calibration is executed tocorrect the difference between the eyes of individuals (the size, thecurvature radius, etc. of the eyeball), and the difference between theimaging devices (specifically, the position relationship between theimaging device and the display screen image, etc).

Here, the calibration corrects the error by instructing the user to lookat a specific site and recording the state of the eye at that time. Thiscalibration adjusts the error between the intersection point of theposition and the direction (the visual axis) of the sight line on thedisplay screen image and the point that the user looks at on the displayscreen image. The error arises from the difference of the eye of theindividual the difference of the imaging device.

FIG. 3 is a schematic diagram for describing the calibration accordingto a comparative example. In the comparative example illustrated in FIG.3, five gazing points are sequentially displayed one after another onthe display screen of the display unit 11, to prompt the user to gaze atthe gazing points P1 to P5. During this, the user gazes for about onesecond at each gazing point.

FIG. 4 is a flowchart illustrating a procedure of the calibrationaccording to the comparative example. The flowchart of FIG. 4 startsfrom displaying the gazing point (for example, the gazing point P1)illustrated in FIG. 3 (step S902). Next, whether or not the user gazesat the gazing point P1 is determined (step S904). Here, whether the usergazes is decided based on whether or not the eye remains still in theshot image capturing the user by the camera, for example.

If it is determined that the user gazes in step S904 (Yes), thecoordinates of the eye in the shot image, and the coordinates of thegazing point P1 in the display unit (which is preset) are paired andrecorded (step S906). Then, in order to record the coordinates of theeye relative to all gazing points P1 to P5 (step S908: No), the sameprocessing is executed after proceeding to the next gazing point (stepS910).

On the other hand, if the coordinates of the eye relative to each of thegazing points P1 to P5 are recorded (step S908: Yes), the correspondencerelationship between the coordinates of the eye in the shot image andthe coordinates of the gazing point in the display screen image iscalculated (step S912). Then, the error is corrected on the basis of thecalculation result.

In the meantime, in the comparative example described above, since theuser gazes at each of the five gazing points for one second, the usergazes for at least five seconds. Hence, this is a psychological andtemporal burden for the user.

Also, in order to improve the detection accuracy of the sight line,increasing the number of the gazing points, gazing again the gazingpoint for which the camera has failed to adequately capture the eye ofthe user, and repeatedly executing the calibration itself are taken asmeasures. However, when these measures are taken, the time used for thecalibration is further prolonged, and the psychological and temporalload given to the user is increased.

In contrast, the information processing apparatus according to anembodiment of the present disclosure described in the following executesthe calibration during the normal user interface (UI) operation of theuser who is the operator, to reduce the psychological and temporal loadgiven to the user. That is, the calibration is executed in such a mannerthat the user does not become conscious of the burden.

2. First Embodiment

(2-1. Configuration of Information Processing Apparatus)

With reference to FIG. 5, description will be made of an example of theconfiguration of the information processing apparatus according to anembodiment of the present disclosure. FIG. 5 is a block diagramillustrating an example of the function and configuration of theinformation processing apparatus according to an embodiment of thepresent disclosure.

In the present embodiment, the information processing apparatus 100 isequipped in the information device such as a notebook personal computer(PC) (here, the information device 10 illustrated in FIG. 1). Asillustrated in FIG. 5, the information processing apparatus 100 includesa display control unit 108, an operation detecting unit 110, a sightline detecting unit 112, a correlation acquiring unit 114, a regionestimating unit 116, a correction coefficient acquiring unit 118, and anoperation switching unit 120.

(Display Control Unit 108)

The display control unit 108 controls the display of the display unit 11(FIG. 1) of the information device 10. The display unit 11 displays thedisplay screen image for the UI operation, to which the operator such asthe user performs the input operation, for example. Note that thedisplay unit 11 is provided integrally with the information device 10,but is not limited thereto. The display unit 11 may be a configurationseparated from the information device 10.

FIG. 6 is a diagram illustrating an example of the display screen imagefor the UI operation including operation objects. In the display screenimage 12 illustrated in FIG. 6, the cursor and the icon are displayed asthe operation objects. Then, the user operates the cursor and the icon(specifically, presses the icon by an input interface (for example, amouse or a touch pad)), in order to execute the corresponding function(a predetermined input).

Also, the display control unit 108 adds an animation to the operationobject that is to be operated by the operator. Here, the added animationexpression are, for example, an animation expression by brightnesschange such as lighting and blinking of the cursor or the icon which arethe operation objects, an animation expression by size change of thecursor and the icon, and an animation expression by color change of thecursor and the icon. By adding these animations, the operator cannaturally continue looking at the operation object displayed in thedisplay screen image 12. As a result, it becomes easy to acquire thedata for calibration.

(Operation Detecting Unit 110)

The operation detecting unit 110 detects the operation of the operatorto the display screen image 12. For example, the operation detectingunit 110 detects the operation of the operator to the operation objectthat is displayed in the display screen image for executing apredetermined input. The operation detecting unit 110 acquires theinformation from an input interface such as a mouse operated by theoperator, in order to detect the operation of the operator to theoperation object.

Note that, the operation of the operation object is not limited to theinput interface. For example, the operation of the operation object maybe a camera gesture, and may be an operation by the sight line.

(Sight Line Detecting Unit 112)

The sight line detecting unit 112 detects the movement of the sight lineof the operator on the display screen image. For example, the imagingunit 13 (FIG. 1) equipped in the information device 10 captures an imageof the operator, so that the sight line detecting unit 112 detects themovement of the sight line of the operator.

(Correlation Acquiring Unit 114)

The operator normally performs the operation while looking at the cursoror the icon which are the operation objects. That is, the sight line ofthe operator gazes at the cursor or the icon. Therefore, in the presentembodiment when it is determined that the operator continues looking at(following with the eye) the operation object displayed in the displayscreen image 12, the data for the calibration is employed.

In the present embodiment, in order to determine whether the operatorcontinues looking at the operation object, the correlation between thefirst transition information (time-series data) indicating thetransition of the movement of the operation object, and the secondtransition information (time-series data) indicating the transition ofthe movement of the sight line of the operator is checked as describedin the following.

The correlation acquiring unit 114 acquires the correlation between thefirst transition information of the operation object operated by theoperator, and the second transition information of the sight line duringthe operation of the operator. Then, in order to determine thecorrelation, the correlation acquiring unit 114 combines the firsttransition information and the second transition information tocalculate a cross-correlation function.

Here, description will be made of the cross-correlation function of thetwo types of time-series data, with reference to FIGS. 7 and 8.

FIG. 7 is a diagram for describing the cross-correlation function whenthe waveforms of the two time-series data functions are similar to eachother. FIG. 8 is a diagram for describing the cross-correlation functionwhen the waveforms of the time-series data functions are not similar toeach other. The fact that the waveforms of the two time-series datafunctions are similar to each other means that the operator follows theoperation object with the eye. The fact that the waveforms of the twotime-series data functions are not similar to each other means that theoperator does not follow the operation object with the eye. Note that,even when the operator follows the operation object with the eye, adifference in phase is generated in the two waveforms of the time-seriesdata.

Description will be made of how to calculate the cross-correlationfunction according to the present embodiment. Here, two types oftime-series data are expressed by the function f(a) and the functionh(a), respectively. In FIG. 7, the function f(a) and the function h(a)have a relationship that is different in phase and similar in waveform.On the other hand, in FIG. 8, the function f(a) and the function h(a)have a relationship that is different in phase and not similar inwaveform.

First, only one of the two functions is shifted by x. Here, the functionh(a) is shifted by x. Next, the product of the function h(a−x) and thefunction f(x) is calculated. With additional integration, thecross-correlation function g(x) like the below formula is obtained.g(x)=∫_(−∞) ^(∞) h(a−x)f(a)da

The obtained cross-correlation function g(x) has the below feature. Thatis, when the waveforms of the two time-series data functions are similarto each other, the maximum value of the cross-correlation function g(x)is larger than a predetermined threshold value Tcc as illustrated inFIG. 7. On the other hand, when the waveforms of the two time-seriesdata functions are not similar to each other, the maximum value of thecross-correlation function g(x) is smaller the predetermined thresholdvalue Tcc as illustrated in FIG. 8.

Hence, when the calculated maximum value of the cross-correlationfunction is equal to or larger than the predetermined threshold value,the correlation acquiring unit 114 determines that the correlation ofthe first transition information and the second transition informationis close. On the other hand, when the calculated maximum value of thecross-correlation function is smaller than the predetermined thresholdvalue, the correlation acquiring unit 114 determines that thecorrelation of the first transition information and the secondtransition information is not close. Thereby, the mutual relationship isacquired accurately.

Then, when a certain number or more of data points for the calibrationare acquired, the calibration parameter is calculated. Hence, in thepresent embodiment, while the user operates the operation objectdisplayed in the display screen image, the calibration is executed inthe background.

Although in the above the cross-correlation function of the two types oftime-series data is calculated, the present disclosure is not limitedthereto. For example, the correlation acquiring unit 114 may use across-correlation function normalized as the cross-correlation function.In that case, the predetermined threshold value Tcc is set within arange from −1 to 1.

(Region Estimating Unit 116)

The region estimating unit 116 estimates an attention region, to whichthe operator pays attention, in the display screen image. The regionestimating unit 116 outputs the information of the estimated attentionregion to the correlation acquiring unit 114. Then, the correlationacquiring unit 114 acquires the correlation between the first transitioninformation operated by the operator in the attention region, and thesecond transition information of the sight line during the operation ofthe operator. Thereby, only the operation objects within the region towhich the user can easily pay attention are estimated among a pluralityof the displayed operation objects, and the operation objects within theattention region are subjected to calculation. As a result, the numberof the operation objects for which the time-series data of the positionare recorded is reduced.

Here, an example of the method to estimate the attention region of theuser utilizing a publicly known saliency map will be described.

The saliency map is a map that represents the intensity of saliencycorresponding to the input image. Then, the saliency map represents thedegrees of “interest of a person” with respect to an image regions, andis calculated by combining image features such as an edge and a color.For example, in the saliency map with respect to the input imagecapturing a ship floating on the sea, the part of the ship is emphasized(for example, the part of the ship is emphasized by making it white inan solid black image).

FIG. 9 is a diagram illustrating an example of the saliency map withrespect to a display screen image including operation objects. In FIG.9, the input image includes three icons and one cursor, which are theoperation objects. In that case, the part corresponding to the threeicons and one cursor are presented in an emphasized manner in thesaliency map. Thereby, the information processing apparatus 100estimates the parts emphasized in the saliency map as the region towhich the user pays attention, and records the time-series data of theposition of the operation objects in the estimated attention region. Asa result, the time-series data of the position of the operation objectof the region to which the user does not pay attention is not recorded.

(Correction Coefficient Acquiring Unit 118)

The correction coefficient acquiring unit 118 acquires the correctioncoefficient for correcting the error of the sight line input that theoperator performs, on the basis of the movement of the sight linedetected during the operation of the operator to the operation object.The acquired correction coefficient is the coefficient for correctingthe error and for the calibration. Thereby, in the present embodiment,the calibration is executed in the background, while the user operatesthe operation object displayed in the display screen image.

The correction coefficient acquiring unit 118 acquires the correctioncoefficient on the basis of the correlation acquired by the correlationacquiring unit 114. That is, the correction coefficient is acquired onthe basis of the correlation between the movement of the operationobject and the movement of the sight line.

At this, when the correlation between the first transition informationof the operation object and the second transition information of thesight line is close, the correction coefficient acquiring unit 118 setsthe first transition information and the second transition information,as the data for acquiring the correction coefficient. When thecorrelation of the first transition information and the secondtransition information is not close, the correction coefficientacquiring unit 118 does not set the first transition information and thesecond transition information, as the data for acquiring the correctioncoefficient. Thereby, only the first transition information and thesecond transition information between which the correlation is close areutilized as the data for the calibration.

(Operation Switching Unit 120)

The operation switching unit 120 switches the input operation by theoperator, depending on the data number of the first transitioninformation and the second transition information effective foracquiring the correction coefficient. That is, the operation switchingunit 120 switches the input operation by the operator, depending on thedata number for the calibration acquired in the background during theoperation of the operation object.

Here, the input operation by the camera gesture and the input operationby the sight line will be described as an example. The operationswitching unit 120 switches to the input operation by the gesture whenthe data number for the calibration is small, and switches to the inputoperation by the sight line when the data number for the calibration islarge.

At this, a graphical user interface (GUI) for informing the user thatthe input operation is executable by the sight line may be displayed inresponse to the state of the input operation. Also, the input interfacethat the user can utilize may be displayed. Thereby, the usability isimproved. Note that, when the accuracy of the calibration is high, thegesture and the sight line may be combined, in such a manner that theobject to which the input operation is performed by the gesture isselected by the sight line. As a result, the intention of the user isreflected more highly accurately.

Note that, in above, the information processing apparatus 100 isequipped in the notebook PC which is the information device, but is notlimited thereto. For example, the information processing apparatus 100may be equipped in a device such as a smartphone, a tablet, a gamemachine, a television, and a digital signage.

(2-2. Operation of Information Processing Apparatus)

With reference to FIG. 10, description will be made of an example of theoperation of the information processing apparatus 100 having theconfiguration described above. FIG. 10 is a flowchart illustrating anexample of the operation of the information processing apparatus 100according to an embodiment of the present disclosure.

The process illustrated in FIG. 10 is realized by the CPU of theinformation processing apparatus 100 which executes a program stored inthe ROM. Note that the executed program may be stored in a recordingmedium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), amemory card, or may be downloaded from a server or the like via theInternet.

The flowchart of FIG. 10 is started from displaying the display screenimage 12 (FIG. 6) for UI operation, including the operation object. Theuser operates the operation object in the display screen image 12, toexecute a desired function by the input interface, for example.

The information processing apparatus 100 first acquires input datacorresponding to the user operation from the input interface (stepS102). Next, the information processing apparatus 100 updates thedisplay position of the operation object in response to the input data,and saves the time-series data D1 of the position within a predeterminedtime of the past (step S104).

Next, the information processing apparatus 100 acquires the position andthe direction of the eyeball by the sight line detecting unit (stepS106). Next, the information processing apparatus 100 determines whetheror not the user's calibration parameter of the past is saved (stepS108).

If the calibration parameter of the past is saved in step S108 (Yes),the information processing apparatus 100 estimates the point of thedisplay screen image that the user looks at, by the calibrationparameter of the past. Then, the information processing apparatus 100saves the time-series data D2 within a predetermined time of the past atthe estimated point (step S110).

On the other hand, if the calibration parameter of the past is not savedin step S108 (No), the information processing apparatus 100 estimatesthe point of the display screen image that the user looks at by thedefault calibration parameter. Then, the information processingapparatus 100 saves the time-series data D2 within a predetermined timeof the past at the estimated point (step S112).

Next, the information processing apparatus 100 calculates thecross-correlation function (FIG. 7) between the time-series data D1 andthe time-series data D2. Then, the information processing apparatus 100determines whether or not the condition that the calculated maximumvalue of the cross-correlation function is equal to or larger than thepredetermined threshold value and continues to be equal to or largerthan the predetermined threshold value for a predetermined time or moreis satisfied (step S114).

If the condition that the maximum value of the cross-correlationfunction is equal to or larger than the a predetermined threshold valueand continues to be equal to or larger than the predetermined thresholdvalue for a predetermined time or more is not satisfied in step S114(No), the information processing apparatus 100 repeatedly executes theprocess of step S102 to S112 described above.

On the other hand, if the condition that the maximum value of thecross-correlation function is equal to or larger than the apredetermined threshold value and continues to be equal to or largerthan the predetermined threshold value for a predetermined time or moreis satisfied in step S114 (Yes), the information processing apparatus100 records the position of the operation object, and the estimatedpoint of the display screen image, as the data for calibration (stepS116).

Next, the information processing apparatus 100 determines whether or nota predetermined number or more of data pieces for calibration arerecorded (step S118). Then, if the predetermined number or more of thedata pieces for calibration are recorded in step S118 (Yes), theinformation processing apparatus 100 calculates the correspondencerelationship with the point in the display screen image that the userlooks at, using the data for calibration (step S120).

On the other hand, if a predetermined number or more of the data piecesfor calibration are not recorded in step S118 (No), the informationprocessing apparatus 100 continues searching for the data for the nextcalibration (step S122), and repeats the process described above.

3. Second Embodiment

In the meantime, in the calculation of the cross-correlation functiondescribed above, when the time-series data of the position of theoperation object and the point on the display screen image that the userlooks at do not change, the maximum value of the cross-correlationfunction is high regardless of the position on the display screen image.Hence, when a plurality of operation objects are displayed in thedisplay screen image, the increased number of the operation objects thatis to be calculated makes it difficult to appropriately select theoperation object that is to be calculated, which generates a problemthat the operation object that the user has actually looked at is notcalculated.

Therefore, in order to solve the above problem, in the secondembodiment, the information processing apparatus 100 executes thecalculation for the operation object having a distance equal to orshorter than a predetermined threshold value from the sight lineposition of the user, among a plurality of the operation objects, anddoes not execute the calculation for the operation object having adistance longer than the predetermined threshold value from the sightline position.

Specifically, the correlation acquiring unit 114 (FIG. 5) of theinformation processing apparatus 100 calculates the distance between thesight line position of the sight line on the display screen image 12 andthe gravity center position of the operation object. When the calculateddistance is equal to or shorter than the predetermined threshold value,the correlation acquiring unit 114 calculates the cross-correlationfunction using the first transition information of the operation object.Thereby, the calculation is executed for the operation object having ahigh possibility that the user actually looks at.

FIG. 11 is a diagram illustrating the distance between the sight lineposition and the gravity center position of the operation object. InFIG. 11, among a plurality of icons displayed in the display screenimage, the calculation is executed for the icon 3 only, which has adistance equal to or shorter than the predetermined threshold value fromthe unmoving sight line position.

Also, the information processing apparatus 100 does not execute thedetermination of step S114 of FIG. 10, when the change of the positionof the operation object is approximately zero, that is, when the totalmoving amount of the operation object in the time-series data is equalto or shorter than a certain threshold value. Hence, it is desirablethat the determination of whether or not the total moving amount of theoperation object is equal to or shorter than the threshold value beexecuted immediately before step S114 of FIG. 10.

Specifically, the correlation acquiring unit 114 does not calculate thecross-correlation function using the first transition information of theoperation object, when the moving amount of the operation object isequal to or shorter than the predetermined threshold value. Thereby, thecalculation is not executed to the unmoving operation object whoseposition does not change.

Although in the above the calibration in the sight line detection hasbeen described, the above may be applied to the calibration in thefinger pointing. When the input operation by the sight line and theinput operation by the finger pointing are combined to be utilized, itmay be determined whether one operation is prioritized in considerationof the accuracies of the both calibrations. Also, in estimating theposition pointed by the user on the display screen image, each estimateresult may be multiplied by a coefficient according to the calibrationaccuracy, in order to execute estimation.

4. Hardware Configuration

The operation by the information processing apparatus 100 describedabove is realized by the cooperation of the hardware configuration andthe software of the information processing apparatus 100.

FIG. 12 is an explanatory diagram illustrating the exemplary hardwareconfiguration of the information processing apparatus 100. Asillustrated in FIG. 12, the information processing apparatus 100includes a CPU (Central Processing Unit) 801, a ROM (Read Only Memory)802, a RAM (Random Access Memory) 803, an input device 808, an outputdevice 810, a storage device 811, a drive 812, an imaging device 813,and a communication device 215.

The CPU 801 functions as an operation processor and a control device,and controls the overall operation of the information processingapparatus 100 in accordance with various types of programs. Also, theCPU 801 may be a microprocessor. The ROM 802 stores programs, operationparameters, and other data used by the CPU 801. The RAM 803 temporarilystores the programs used in the execution of the CPU 801, the parametersthat change as appropriate in the execution of the programs, and otherdata. They are connected to each other by a host bus configured from aCPU bus and others.

The input device 808 is composed of a mouse, a keyboard, a touch panel,a button, a microphone, input means for the user to input informationsuch as a switch and a lever, an input control circuit that generates aninput signal on the basis of input by the user and outputs the inputsignal to the CPU 801, and others. The user of the informationprocessing apparatus 100 operates the input device 808, in order toinput the various types of data to the information processing apparatus100 and instruct the processing operation.

The output device 810 includes a display device, such as for example aliquid crystal display (LCD) device, an organic light emitting diode(OLED) device, and a lamp. Further, the output device 810 includes anaudio output device such as a speaker and a headphone. For example, thedisplay device displays a captured image, a generated image, and thelike. On the other hand, the audio output device converts sound data tosound and outputs the sound.

The storage device 811 is a device for data storage which is configuredas one example of the storage unit of the information processingapparatus 100 according to the present embodiment. The storage device811 may include a storage medium, a recording device that records dataon a storage medium, a reading device that reads out data from a storagemedium, a deleting device that deletes data recorded on a storagemedium, and a like. The storage device 811 stores programs and varioustypes of data executed by the CPU 801.

The drive 812 is a storage medium reader/writer, which is providedeither inside or outside the information processing apparatus 100. Thedrive 812 reads out the information recorded on a removable storagemedium 820 such as a magnetic disk, an optical disc, a magneto-opticaldisk, or a semiconductor memory mounted thereon, and output to the RAM803. Also, the drive 812 is capable of writing information on theremovable storage medium 820.

The imaging device 813 includes an imaging optical system such as aphotographing lens and a zoom lens that condenses light and a signalconversion element such as a charge coupled device (CCD) and acomplementary metal oxide semiconductor (CMOS). The imaging opticalsystem condenses light emitted from a subject to form an image of thesubject on a signal conversion unit. The signal conversion elementconverts the formed image of the subject into an electric image signal.

The communication device 815 is, for example, a communication interfaceconfigured by a communication device for connecting to the network 830and other devices. Also, the communication device 815 may be a wirelessLAN (Local Area Network) compatible communication device, a LTE (LongTerm Evolution) compatible communication device, or a wire communicationdevice that communicates via wire.

Note that, the network 830 is a wired or wireless transmission channelof the information transmitted from a device connected to the network830. For example, the network 830 may include public line networks suchas the Internet, a telephone line network, a satellite communicationnetwork, various types of local area networks (LAN) including theEthernet (registered trademark), wide area networks (WAN), and others.Also, the network 830 may include dedicated line networks such as IP-VPN(Internet Protocol-Virtual Private Network).

5. Conclusion

The information processing apparatus 100 described above acquires thecorrection coefficient for correcting the error of the sight line inputthat the operator performs, on the basis of the movement of the sightline of the operator detected during the operation of the operator tothe operation object (a cursor, an icon, etc) displayed in the displayscreen image 12. That is, the information processing apparatus 100executes the sight line calibration in the background, during theoperation of the normal operation object by the operator.

Thereby, the gazing point (FIG. 2) for the sight line calibration isneedless to be displayed, the operator is needless to move the sightline in accordance with the display of the gazing point. Accordingly,the calibration for improving the sight line tracking accuracy isexecuted without making the user conscious of the load.

Note that, in the above, the information processing apparatus 100 isequipped in the information device 10 having the display screen image12, but is not limited thereto. As illustrated in FIG. 13, theinformation processing apparatus 100 may be provided in the server 50capable of communicating with the information device via a network.

FIG. 13 is a diagram illustrating an example of the function andconfiguration of the information processing apparatus provided in theserver. The server 50 illustrated in FIG. 13 includes an informationprocessing apparatus 100 in the same way as the configuration describedin FIG. 5, and a communication unit 52 capable of communicating with theinformation device 10. When the operator operates the operation objectdisplayed in the display screen image 12 of the information device 10,the transition information of the movement of the operation object, andthe transition information of the movement of the sight line of theoperator are transmitted to the server 50 via the communication unit 15.Thereby, the server 50 acquires the correction coefficient forcorrecting the error of the sight line input that the operator performs.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

The steps illustrated in the flowcharts in the above-describedembodiment naturally include processes performed in the described andchronological order, and further include processes that are notnecessarily performed in chronological order, but are also performed inparallel or are individually performed. It is also possible to changethe order as necessary even in the steps for chronologically performingthe processes.

A process performed by the information processing apparatus described inthe present specification may be realized by using any one of software,hardware, and a combination of software and hardware. A program includedin software is stored in advance in, for example, a storage medium thatis built in or externally provided to each apparatus. When executed,programs are each read out by, for example, Random Access Memory (RAM),and executed by a processor such as a CPU.

Additionally, the present technology may also be configured as below.

(1) An information processing apparatus including:

an operation detecting unit configured to detect an operation of anoperator to an object that is displayed in a display screen image forperforming a predetermined input;

a sight line detecting unit configured to detect a movement of a sightline of the operator on the display screen image; and

a correction coefficient acquiring unit configured to acquire acorrection coefficient for correcting an error of a case where theoperator performs sight line input, on the basis of the movement of thesight line detected during the operation of the operator to the object.

(2) The information processing apparatus according to (1), furtherincluding a correlation acquiring unit configured to acquire acorrelation between first transition information of the object operatedby the operator and second transition information of the sight lineduring the operation of the operator,

wherein the correction coefficient acquiring unit acquires thecorrection coefficient on the basis of the acquired correlation.

(3) The information processing apparatus according to (2), wherein thecorrection coefficient acquiring unit

sets the first transition information and the second transitioninformation, as data for acquiring the correction coefficient, when thecorrelation between the first transition information of the object andthe second transition information of the sight line is close, anddoes not set the first transition information and the second transitioninformation, as data for acquiring the correction coefficient, when thecorrelation between the first transition information and the secondtransition information is not close.

(4) The information processing apparatus according to (3), wherein thecorrelation acquiring unit

combines the first transition information and the second transitioninformation to calculate a cross-correlation function,

determines that the correlation between the first transition informationand the second transition information is close, when a maximum value ofthe calculated cross-correlation function is equal to or larger than apredetermined threshold value, and

determines that the correlation between the first transition informationand the second transition information is not close, when the maximumvalue of the cross-correlation function is smaller than thepredetermined threshold value.

(5) The information processing apparatus according to (4), wherein thecorrelation acquiring unit uses a normalized cross-correlation functionas the cross-correlation function.

(6) The information processing apparatus according to (4) or (5),wherein the correlation acquiring unit

calculates a distance between a sight line position of the sight line onthe display screen image and a gravity center position of the object,and

calculates the cross-correlation function using the first transitioninformation of the object, when the distance is equal to or shorter thanthe predetermined threshold value.

(7) The information processing apparatus according to (4) or (5),wherein the correlation acquiring unit does not calculate thecross-correlation function using the first transition information of theobject, when a moving amount of the object is equal to or shorter than apredetermined threshold value.

(8) The information processing apparatus according to any one of (1) to(7), further including

a display control unit configured to add an animation to the object thatis to be operated by the operator.

(9) The information processing apparatus according to any one of (2) to(7), further including

a region estimating unit configured to estimate a region to which theoperator pays attention in the display screen image,

wherein the correlation acquiring unit acquires a correlation betweenthe first transition information of the object operated by the operatorin the estimated region and the second transition information of thesight line during the operation of the operator.

(10) The information processing apparatus according to any one of (3) to(7), further including

an operation switching unit configured to switch an input operation bythe operator, depending on data numbers of the first transitioninformation and the second transition information that are effective foracquiring the correction coefficient.

(11) An information processing method including:

detecting an operation of an operator to an object that is displayed ina display screen image for performing a predetermined input;

detecting a movement of a sight line of the operator on the displayscreen image; and acquiring a correction coefficient for correcting anerror of a case where the operator performs sight line input, on thebasis of the movement of the sight line detected during the operation ofthe operator to the object.

(12) A program for causing a computer to execute:

detecting an operation of an operator to an object that is displayed ina display screen image for performing a predetermined input;

detecting a movement of a sight line of the operator on the displayscreen image; and

acquiring a correction coefficient for correcting an error of a casewhere the operator performs sight line input, on the basis of themovement of the sight line detected during the operation of the operatorto the object.

What is claimed is:
 1. An information processing apparatus, comprising:at least one processor configured to: acquire, from an input interface,operating information on an operation of a user to a first object and asecond object that are displayed at a first position and a secondposition on a display screen respectively, wherein the first position isdifferent from the second position; acquire from a sight-line detector,sight-line movement information on a movement of a sight line of theuser from the first object to the second object, wherein the sight lineis detected during the operation of the user to the first object and thesecond object; correct sight-line input information based on theoperating information and the sight-line movement information; andcontrol an output device to output the corrected sight-line inputinformation.
 2. The information processing apparatus according to claim1, wherein the at least one processor is further configured to: acquirecorrelation information between the operating information and thesight-line movement information; and acquire a correction coefficientfor correction of the sight-line input information based on the acquiredcorrelation information.
 3. The information processing apparatusaccording to claim 2, wherein the at least one processor is furtherconfigured to correct the sight-line input information based on adetermination that the correlation information is smaller than a firstthreshold value.
 4. The information processing apparatus according toclaim 3, wherein the at least one processor is further configured to:combine the operating information and the sight-line movementinformation to calculate a cross-correlation function, correct thesight-line input information based on a determination that a maximumvalue of the calculated cross-correlation function is equal to or largerthan a second threshold value.
 5. The information processing apparatusaccording to claim 4, wherein the at least one processor is furtherconfigured to use a normalized cross-correlation function as thecross-correlation function.
 6. The information processing apparatusaccording to claim 4, wherein the at least one processor is furtherconfigured to: calculate a distance between a sight line position of thesight line and the first position of the first object on the displayscreen, and correct the sight-line input information based on thedistance that is equal to or shorter than a third threshold value. 7.The information processing apparatus according to claim 4, wherein theat least one processor is further configured not to correct thesight-line input information based on a distance between the firstposition and the second position that is equal to or shorter than afourth threshold value.
 8. The information processing apparatusaccording to claim 1, wherein the at least one processor is furtherconfigured to control the display screen to emphasize the first objectthan other objects displayed on the display screen.
 9. The informationprocessing apparatus according to claim 1, wherein the at least oneprocessor is further configured to: estimate an attention region towhich the user pays attention on the display screen; and select thefirst object and the second object from the estimated attention region.10. The information processing apparatus according to claim 1, whereinthe at least one processor is further configured to: switch an input forthe user between a gesture input and sight-line input based on anaccuracy of the correction of the sight-line input information.
 11. Theinformation processing apparatus according to claim 1, wherein each ofthe first object and the second object is one of a cursor or an iconthat is moved from the first position to the second position through theoperation.
 12. The information processing apparatus according to claim1, wherein the sight-line detector is an imager configured to capture animage including an eye of the user.
 13. An information processingmethod, comprising: acquiring, from an input interface, operatinginformation on an operation of a user to a first object and a secondobject that are displayed at a first position and a second position on adisplay screen respectively, wherein the first position is differentfrom and the second position; acquiring, from a sight-line detector,sight-line movement information on a movement of a sight line of theuser from the first object to the second object, wherein the sight lineis detected during the operation of the user to the first object and thesecond object; correcting sight-line input information based on theoperating information and the sight-line movement information; andcontrolling an output device to output the corrected sight-line inputinformation.
 14. A non-transitory computer-readable medium having storedthereon computer-executable instructions, which when executed by aprocessor, cause the processor to execute operations, the operationscomprising: acquiring, from an input interface, operating information onan operation of a user to a first object and a second object that aredisplayed at a first position and a second position on a display screenrespectively, wherein the first position is different from the secondposition; acquiring, from a sight-line detector, sight-line movementinformation on a movement of a sight line of the user from the firstobject to the second object, wherein the sight line is detected duringthe operation of the user to the first object and the second object;correcting sight-line input information based on the operatinginformation and the sight-line movement information; and controlling anoutput device to output the corrected sight-line input information.